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WWC Review of the Report “Same-Language-Subtitling (SLS): 
Using Subtitled Music Video for Reading Growth ” 1 

The findings from this review do not reflect the full body of research evidence on SLS. 

What is this study about? 

This randomized controlled trial examined the impacts 
of Same-Language-Subtitling (SLS), a karaoke-style 
subtitling intervention, on the reading comprehension 
skills of secondary school students in Kaneohe, Hawaii. 

Researchers randomly assigned 198 secondary school 
students with learning disabilities (ages 14 to 19) to 
either special education classrooms using the SLS 
intervention or comparison classrooms (special or 
general education). The final study sample consisted 
of 51 students in the intervention condition and 98 
students in the comparison condition. 2 

Teachers in the intervention condition used SLS to 
encourage reading and increase reading proficiency 
over a 12-week period. Students engaged in SLS view- 
ing/response activities for 1 5-20 minutes per day at the 
beginning of class, during which they completed cloze- 
style worksheets and responded to comprehension 
questions while viewing the SLS recordings. During the 
last six weeks of the intervention, they also spent a mini- 
mum of 90 minutes per week producing subtitled multi- 
media files. Students in the comparison group received 
the school’s regular “business-as-usual” curriculum. 

Researchers assessed the effectiveness of SLS by 
comparing the reading comprehension achievement of 
students in the SLS intervention and comparison con- 
ditions at the end of the 12-week intervention in June, 
and again after the summer break in September. 3 

What did the study find? 

The study did not report the statistical significance 
of the impact of the SLS intervention. However, 

WWC calculations indicate that students in the SLS 


intervention condition scored significantly higher 
than students in the comparison condition on the 
reading comprehension achievement posttests. 


WWC Rating 


The research described in this 
report meets WWC evidence 
standards without reservations 

Strengths: This study is a well-implemented 
randomized controlled trial. 


Features of Same-Language-Subtitling (SLS) 


SLS is the practice of using videos with a format 
similar to karaoke, where captioned text changes 
color in synchronization with audio. The purpose 
is to encourage reading and increase reading 
proficiency. Students are able to hear the words 
being spoken/sung and read the words being 
spoken/sung at the same time. 

SLS typically uses repetitive sources of audio, such 
as music, poetry, audiobooks, or occasionally, 
famous speeches. In the study described in this 
WWC report, the researchers used audio from three 
Broadway musicals that contained lyrics above the 
students’ reading levels: “Les Miserables,” “Cats,” 
and “Big River.” 

In the final six weeks of the intervention, students 
produced subtitled multimedia files from the “Big 
River” CD during study hall time periods. 

SLS was developed by Dr. Brij Kothari, the president 
of PlanetRead, a non-profit organization dedicated 
to reading and literacy development. It was originally 
used with Bollywood film songs on TV to promote 
literacy in India. 
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Appendix A: Study details 

McCall, W. G., & Craig, C. (2009). Same-Language-Subtitling (SLS): Using subtitled music video for 
reading growth. In G. Siemens & C. Fulford (Eds.), Proceedings of World Conference on Educa- 
tional Multimedia, Hypermedia and Telecommunications 2009 (pp. 3983-3992). Chesapeake, VA: 
AACE. Retrieved from http://www.editlib.Org/p/32055. 


Setting 

The study took place in one school in Kaneohe, Hawaii. 

Study sample 

A sample of 198 students with learning disabilities was chosen to participate in the study 
from a pool of approximately 1,200 students. Sixty-seven students were randomly assigned 
to the SLS classrooms, and the remaining 131 students were assigned to the “business-as- 
usual” comparison condition. Over the course of the study, nine students transferred out of 
the school (two students in the SLS condition and seven in the comparison condition), and 40 
students (14 in the SLS condition and 26 in the comparison condition) were eliminated from 
the impact analysis (these students were characterized as outliers on the baseline assess- 
ments). The final analysis sample included 51 students in SLS classrooms and 98 students in 
comparison classrooms. The students in the analysis sample were between 14 and 19 years 
old and had reading levels ranging between a 2.2 Grade Equivalency (G.E.) and a 9.4 G.E. The 
average reading level of this sample was 5.37 G.E. 

Intervention 

group 

The SLS intervention consisted of two types of SLS music exercises: viewing/response 
activities and producing subtitled multimedia files. During the 12-week intervention, students 
engaged in SLS viewing/response activities for 15-20 minutes per day. This included watching 
three brief repetitions of SLS music videos and, while watching, completing cloze-style work- 
sheets containing questions about the songs (e.g., filling in the blanks of lyrics being sung, or 
reading comprehension based on the lyrics). In some instances, students would echo the sing- 
ing to encourage tracking of the subtitling. 

In the final six weeks of the intervention, students produced subtitled multimedia files from the 
“Big River” CD during study hall time periods. To create the subtitled videos, students used 
Karafun, a free karaoke production program, along with Microsoft Word and PowerPoint. Cre- 
ating the SLS videos took a minimum of 90 minutes per week over the six-week period. 

Comparison 

group 

Students in the comparison group were taught using the existing curriculum in special educa- 
tion and general education classrooms (some of the students with learning disabilities in the 
comparison group were in mainstreamed general education classrooms). Comparison teach- 
ers agreed to use classroom materials that mirrored the difficulty of the materials taught in the 
intervention classrooms (text material that ranged between the fifth- and twelfth-grade reading 
levels). The main difference in the experiences of the intervention and comparison groups was 
exposure to SLS and time spent using computers during the course of the study. 
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Outcomes and 
measurement 


Support for 
implementation 

Reason for 
review 


The study used Accelerated Reader’s Standardized Test and Assessment in Reading (STAR) 
to assess student performance at pretest, immediate posttest (after the 12-week intervention), 
and delayed posttest (after summer break). For a more detailed description of this outcome 
measure, see Appendix B. 

The report did not describe any implementation training or support provided to teachers. 


This study was identified for review by the WWC because it was suggested as a promising 
intervention through the WWC website’s help desk. 
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Appendix B: Outcome measure for the reading comprehension domain 


Reading comprehension 


Standardized Test and Assessment The STAR assessment from the Accelerated Reader system was used as a baseline, immediate, and delayed 

in Reading (STAR) assessment posttest assessment. This assessment is a computer adaptive instrument that is used to identify the reading 

level of a student, measured in terms of grade equivalents. The assessment also produces a scale score, which 
was the score reported by the author in an email correspondence. 
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Appendix C: Study findings for the reading comprehension domain 





Mean 

(standard deviation) 


WWC calculations 


Domain and 
outcome measure 

Study 

sample 

Sample 

size 

Intervention 

group 

Comparison 

group 

Mean 

difference 

Effect 

size 

Improvement 

index 

p-value 

Reading comprehension 

Reading comprehension — 
post-intervention 

Secondary 
school 
students 
ages 14-19 
with learning 
disabilities 

149 

students 

821.7 

(119.45) 

767.0 

(115.57) 

54.70 

0.47 

+18 

0.01 

Reading comprehension — 
delayed post 

Secondary 
school 
students 
ages 14-19 
with learning 
disabilities 

149 

students 

833.7 

(119.94) 

696.0 

(132.23) 

137.70 

1.07 

+36 

0.00 

Domain average for reading comprehension 




0.77 

+28 

Statistically 

significant 


Table Notes: For mean difference, effect size, and improvement index values reported in the table, a positive number favors the intervention group and a negative number favors the 
comparison group. The effect size is a standardized measure of the effect of an intervention on student outcomes, representing the change (measured in standard deviations) in an 
average student’s outcome that can be expected if the student is given the intervention. The improvement index is an alternate presentation of the effect size, reflecting the change in 
an average student’s percentile rank that can be expected if the student is given the intervention. The WWC-computed average effect size is a simple average rounded to two decimal 
places; the average improvement index is calculated from the average effect size. The statistical significance of the study’s domain average was determined by the WWC. The study is 
characterized as having a statistically significant positive effect because the mean effect for the multiple measures of the outcome is positive and statistically significant. 

Study Notes: The intervention and comparison group means and standard deviations in scaled scores were obtained in response to an email request to the authors. The WWC 
calculated the intervention group mean by adding the difference-in-differences adjusted estimate of the average impact of the program (i.e., difference in mean gains between 
the intervention and comparison groups) to the unadjusted comparison group posttests means. Please see the WWC Handbook for more information. The p-value presented here 
was calculated by the WWC, as the author did not conduct inferential tests of the impact of SLS in the article. Both contrasts reported here were determined to be statistically 
significant by the WWC after adjusting for multiple comparisons. 
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Endnotes 

1 Single study reviews examine evidence published in a study (supplemented, if necessary, by information obtained directly from the 
authors]) to assess whether the study design meets WWC evidence standards. The review reports the WWC’s assessment of whether 
the study meets WWC evidence standards and summarizes the study findings following WWC conventions for reporting evidence on 
effectiveness. This study was reviewed using the Students with Learning Disabilities topic area protocol. The WWC rating applies only 
to the results that were eligible under this topic area and met WWC standards without reservations or met WWC standards with reser- 
vations, and not necessarily to all results presented in the study. 

2 One hundred and ninety-eight students were initially randomly assigned to condition — 67 students to special education classrooms 
using the SLS intervention and 131 to comparison special and general education classrooms (this information was obtained in email 
correspondence with the author). A total of nine students left the school before the study completed (two in intervention classrooms 
and seven in comparison classrooms), and 40 other students (14 in the intervention condition and 26 in the comparison condition) 
were removed from the study due to having “unreliable” baseline data (this was a term used by the author to describe students who 
were outliers in the initial baseline assessments). Even if this removal of sample members is considered a source of attrition, the study 
still passes the WWC attrition standard and is therefore eligible for the highest evidence rating as a randomized controlled trial. 

3 In addition to presenting the impacts for the full sample at the end of the intervention and after summer break, the authors presented 
information on program impacts using alternate samples and for subgroups of these separate samples. There was insufficient informa- 
tion to determine if any of these additional analyses met WWC standards. Therefore, this single study report focuses on the impacts 
for the main analysis sample for the two periods of assessment described. 

Recommended Citation 

U.S. Department of Education, Institute of Education Sciences, What Works Clearinghouse. (2013, January). 

WWC review of the report: Same-Language-Subtitling (SLS): Using subtitled music video for reading growth. 
Retrieved from http://whatworks.ed.gov. 
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Glossary of Terms 

Attrition 


Clustering adjustment 
Confounding factor 

Design 
Domain 
Effect size 

Eligibility 

Equivalence 

Improvement index 


Multiple comparison 
adjustment 

Quasi-experimental 
design (QED) 

Randomized controlled 
trial (RCT) 

Single-case design 
(SCD) 

Standard deviation 


Statistical significance 
Substantively important 


Attrition occurs when an outcome variable is not available for all participants initially assigned 
to the intervention and comparison groups. The WWC considers the total attrition rate and 
the difference in attrition rates across groups within a study. 

If intervention assignment is made at a cluster level and the analysis is conducted at the student 
level, the WWC will adjust the statistical significance to account for this mismatch, if necessary. 

A confounding factor is a component of a study that is completely aligned with one of the 
study conditions, making it impossible to separate how much of the observed effect was 
due to the intervention and how much was due to the factor. 

The design of a study is the method by which intervention and comparison groups were assigned. 
A domain is a group of closely related outcomes. 

The effect size is a measure of the magnitude of an effect. The WWC uses a standardized 
measure to facilitate comparisons across studies and outcomes. 

A study is eligible for review if it falls within the scope of the review protocol and uses either 
an experimental or matched comparison group design. 

A demonstration that the analysis sample groups are similar on observed characteristics 
defined in the review area protocol. 

Along a percentile distribution of students, the improvement index represents the gain 
or loss of the average student due to the intervention. As the average student starts at 
the 50th percentile, the measure ranges from -50 to +50. 

When a study includes multiple outcomes or comparison groups, the WWC will adjust 
the statistical significance to account for the multiple comparisons, if necessary. 

A quasi-experimental design (QED) is a research design in which subjects are assigned 
to intervention and comparison groups through a process that is not random. 

A randomized controlled trial (RCT) is an experiment in which investigators randomly assign 
eligible participants into intervention and comparison groups. 

A research approach in which an outcome variable is measured repeatedly within and 
across different conditions that are defined by the presence or absence of an intervention. 

The standard deviation of a measure shows how much variation exists across observations 
in the sample. A low standard deviation indicates that the observations in the sample tend 
to be very close to the mean; a high standard deviation indicates that the observations in 
the sample are spread out over a large range of values. 

Statistical significance is the probability that the difference between groups is a result of 
chance rather than a real difference between the groups. The WWC labels a finding statistically 
significant if the likelihood that the difference is due to chance is less than 5% Ip < 0.05). 

A substantively important finding is one that has an effect size of 0.25 or greater, regardless 
of statistical significance. 


Please see the WWC Procedures and Standards Handbook (version 2.1) for additional details. 
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